Exterior environment recognition device

ABSTRACT

An exterior environment recognition device includes: a specific object detection unit to detect a specific object on the basis of a color image; a data retaining unit to associate and retain the specific object and a luminance range indicating the color of the specific object; and a transparency reduction determination unit to compare a luminance of the color image of the specific object and a luminance range associated with the specific object, and to determine a reduction in transparency of a transparent body located in an image-capturing direction of the onboard camera.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority from Japanese Patent ApplicationNo. 2012-247371 filed on Nov. 9, 2012, the entire contents of which arehereby incorporated by reference.

BACKGROUND

1. Technical Field

The present invention relates to an exterior environment recognitiondevice for recognizing environment outside of a subject vehicle, and,more particularly, relates to an exterior environment recognition devicefor detecting dirt or fog adhered to optical components and thewindshield and appropriately processing an image according to adetection result.

2. Related Art

There have been known exterior environment recognition devices such asone described in Japanese Unexamined Patent Application Publication(JP-A) No. 2010-224925. The exterior environment recognition device hasan onboard camera mounted on a vehicle to capture an image of roadenvironment in front of a subject vehicle, and detects light sourcessuch as a traffic light and a brake lamp on the basis of colorinformation and position information in the image.

When a light source is detected using color information in an image asdescribed above, the following problem may occur: if strong environmentlight such as sunlight (image capturing against the sunlight) is emittedin the image-capturing direction while there is dirt or fog on atransparent body such as a windshield located in the image-capturingdirection and an optical component such as a lens, then, the colorcomponent of the environment light is added to the entire capturedimage, and this causes the target object in the image to have a colordifferent from the original color. As a result, the recognition accuracyof the light source using the color information is reduced, or itbecomes impossible to recognize the light source.

SUMMARY OF THE INVENTION

The present invention is made in view of such problem, and it is anobject of the present invention to provide an exterior environmentrecognition device capable of detecting existence of dirt or fog on thewindshield and an optical component of an onboard camera andappropriately recognizing an image using color information even undersuch environment.

An aspect of the present invention provides an exterior environmentrecognition device for recognizing the environment outside of a subjectvehicle, on the basis of a color image captured by an onboard camera.The exterior environment recognition device includes: a specific objectdetection unit to detect a specific object on the basis of the colorimage; a data retaining unit to associate and retain the specific objectand a luminance range indicating the color of the specific object; and atransparency reduction determination unit to compare a luminance of thecolor image of the specific object and a luminance range associated withthe specific object and to determine a reduction in transparency of atransparent body located in an image-capturing direction of the onboardcamera.

The data retaining unit may further retain and associate the specificobject and the original luminance of the specific object, and theexterior environment recognition device may further include a correctionamount deriving unit to derive the amount of correction on the basis ofa difference between luminance in the color image of the specific objectand the original luminance associated with the specific object, and aluminance correction unit to correct the luminance of the target portionof the color image on the basis of the amount of correction thusderived. The specific object detection unit may detect the specificobject on the basis of the corrected color image.

Another aspect of the present invention provides an exterior environmentrecognition device for recognizing the environment outside of a subjectvehicle, on the basis of a color image captured by an onboard camera.The exterior environment recognition device includes: a specific objectdetection unit to detect a specific object on the basis of the colorimage, a data retaining unit to associate and retain the specific objectand an original luminance indicating the color of the specific object; acorrection amount deriving unit to derive the amount of correction onthe basis of a difference between a luminance in the color image of thespecific object and the original luminance associated with the specificobject; and a luminance correction unit to correct the luminance of thetarget portion of the color image on the basis of the amount ofcorrection thus derived. The specific object detection unit detects thespecific object on the basis of the corrected color image.

The exterior environment recognition device may further include atransparency reduction detection unit to detect a reduction in thetransparency of the transparent body located in an image-capturingdirection of the onboard camera. The correction amount deriving unitderives the amount of correction when the reduction in the transparencyof the transparent body is detected.

The correction amount deriving unit may derive a basic amount ofcorrection obtained by dividing the derived amount of correction by anexposure time of the color image, and the luminance correction unit maycorrect the luminance of the target portion of the color image on thebasis of the amount of correction obtained by multiplying the basicamount of correction by the exposure time of the color image ofcorrection target.

The correction amount deriving unit may derive the amount of correctionper divided region obtained by dividing the color image into multipleregions.

The correction amount deriving unit may derive an amount of correctionof a divided region for which the amount of correction is not derived,on the basis of an amount of correction of a divided region for which anamount of correction is derived.

The correction amount deriving unit may adopt again, as the amount ofcorrection, a time average value of the amount of correction derived onthe basis of a difference between luminance of the color image of thespecific object and the original luminance associated with the specificobject and the amount of correction previously derived in the samedetection region or the same divided region.

The specific object detection unit may detect the specific object on thebasis of a temporal luminance change of the color image over time.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a connection relationship in anenvironment recognition system;

FIGS. 2A and 2B are explanatory diagrams for explaining a luminanceimage and a distance image;

FIG. 3 is an explanatory diagram for explaining an effect of environmentlight in the environment recognition system;

FIG. 4 is a functional block diagram schematically illustratingfunctions of the exterior environment recognition device;

FIG. 5A is an explanatory diagram for explaining a specific objectcorrespondence table;

FIG. 5B is an explanatory diagram for explaining a dirt determinationtable;

FIG. 5C is an explanatory diagram for explaining a correction referencetable;

FIG. 6 is an explanatory diagram for explaining conversion intothree-dimensional position information by a position informationobtaining unit;

FIG. 7 is an explanatory diagram for explaining a specific object map;

FIG. 8 is an explanatory diagram for explaining a determination targetof a transparency reduction determination unit;

FIG. 9 is an explanatory diagram for explaining processing of acorrection amount deriving unit;

FIG. 10 is another explanatory diagram for explaining processing of thecorrection amount deriving unit;

FIG. 11 is a flowchart illustrating an overall flow of an environmentrecognition method;

FIG. 12 is a flowchart illustrating a flow of specific object mapgenerating processing;

FIG. 13 is a flowchart illustrating a flow of grouping processing;

FIG. 14 is a flowchart illustrating a flow of specific objectdetermining processing;

FIG. 15 is a flowchart illustrating a flow of transparency reductiondetermination processing; and

FIG. 16 is a flowchart illustrating a flow of correction amount derivingprocessing.

DETAILED DESCRIPTION

An example of the present invention will be hereinafter explained indetail with reference to attached drawings. The size, materials, andother specific numerical values illustrated in the example are merelyexemplification for the sake of easy understanding of the invention, andunless otherwise specified, they do not limit the present invention. Inthis specification and the drawings, elements having substantially thesame functions and configurations are denoted with same referencenumerals, and repeated explanation thereabout is omitted. Elements notdirectly related to the present invention are omitted in the drawings.

(Environment Recognition System 100)

FIG. 1 is a block diagram illustrating a connection relationship in anenvironment recognition system 100. The environment recognition system100 includes multiple image capturing devices 110 (in this example, twoimage capturing devices 110) serving as onboard cameras, an imageprocessing device 120, an exterior environment recognition device 130,and a vehicle control device 140, which are provided in a vehicle 1.

The image capturing device 110 includes an image element such as acharge-coupled device (CCD) and a complementary metal-oxidesemiconductor (CMOS), and can obtain a color image, that is, luminanceof three color phases (red: R, green: G, blue: B) per pixel. In thisexample, when color and luminance are treated equally, and both termsare included in the same text, they can be read luminance constitutingcolor or color having luminance. In this case, color images taken by theimage capturing devices 110 are referred to as luminance images, whichare distinguished from a distance image to be explained later.

The image capturing devices 110 are disposed to be spaced apart fromeach other in a substantially horizontal direction so that optical axesof the two image capturing devices 110 are substantially parallel in aproceeding direction of the vehicle. The image capturing device 110continuously generates image data obtained by capturing an image of atarget object existing in front of the vehicle 1 on every frame at every1/60 seconds (60 fps), for example. In this case, the target object maybe not only independent three-dimensional objects such as a vehicle, atraffic light, a road, and a guardrail, but also illuminating portionssuch as a tail lamp, a turn signal, a traffic light that can bespecified as portions of three-dimensional objects. Each later-describedfunctional unit in the example below performs processing for each framein response to the update of such image data.

The image processing device 120 obtains image data from each of the twoimage capturing devices 110, derives parallax using so-called patternmatching for searching a block corresponding to any block extracted fromone of the image data pieces, from the other image data. The block is,for example, an array including four pixels in the horizontal directionand four pixels in the vertical direction). In this embodiment, the“horizontal” means the horizontal direction of the captured image andcorresponds to the width direction inte real world, and the “vertical”means the vertical direction of the captured image, and corresponds theheight direction in the real world.

One way of performing the pattern matching is to compare luminancevalues (Y color difference signals) between two pieces of image data perblock indicating any image position. Examples include a Sum of AbsoluteDifference (SAD) obtaining a difference of luminance values, a Sum ofSquared intensity Difference (SSD) squaring a difference and using thesquared difference, and a Normalized Cross Correlation (NCC) adoptingthe degree of similarity of dispersion values obtained by subtracting aluminance mean value from a luminance value of each pixel. The imageprocessing device 120 performs such parallax deriving processing perblock on all the blocks appearing in the detection region (for example,600 pixels by 180 pixels). In this case, the block is assumed to include4 pixels×4 pixels, but the number of pixels in the block may be set atany value.

Although the image processing device 120 can derive a parallax for eachblock serving as a detection resolution unit, it is impossible torecognize what kind of target object the block belongs to. Therefore,the parallax information is not derived per target object, but isindependently derived in detection resolution units (for example, perblock) in the detection region. In this embodiment, an image obtained byassociating the parallax information thus derived (corresponding to alater-described relative distance) with image data is referred to as adistance image.

FIGS. 2A and 2B are explanatory diagrams for explaining a luminanceimage 124 and a distance image 126. For example, assume that theluminance image (image data) 124 as illustrated in FIG. 2A is generatedwith regard to a detection region 122 by the two image capturing devices110. Here, for the sake of easy understanding, only one of the twoluminance images 124 is schematically illustrated. In the this example,the image processing device 120 obtains a parallax for each block fromsuch luminance image 124, and forms the distance image 126 asillustrated in FIG. 2B. Each block of the distance image 126 isassociated with a parallax of the block. In the drawing, for the sake ofexplanation, a block from which a parallax is derived is indicated by ablack dot.

The parallax can be easily specified at an edge portion (portion wherethere is a large contrast difference between adjacent pixels) ofobjects, and therefore, the block from which parallax is derived, whichis denoted with black dots in the distance image 126, is likely to alsobe an edge in the luminance image 124. Therefore, the luminance image124 as illustrated in FIG. 2A and the distance image 126 as illustratedin FIG. 2B are similar in terms of outline of each target object.

Back to FIG. 1, the exterior environment recognition device 130 obtainsthe luminance image 124 and the distance image 126 from the imageprocessing device 120, and uses the luminance based on the luminanceimage 124 to determine which specific object the target object in thedetection region 122 corresponds to. In addition, a relative distance ofthe vehicle 1 based on the distance image 126 is also used to identifythe target object. In this example, the exterior environment recognitiondevice 130 uses a so-called stereo method to convert the parallaxinformation for each block in the detection region 122 of the distanceimage 126 into three-dimensional position information including arelative distance. The stereo method is a method using a triangulationmethod to derive a relative distance of a target object with respect tothe image capturing device 110 from the parallax of the target object.The exterior environment recognition device 130 will be explained laterin detail.

The vehicle control device 140 avoids a collision with the target objectspecified by the exterior environment recognition device 130 andperforms control so as to maintain a safe distance from the precedingvehicle. More specifically, the vehicle control device 140 obtains acurrent cruising state of the vehicle 1 based on, for instance, asteering angle sensor 142 for detecting an steering angle and a vehiclespeed sensor 144 for detecting the speed of the vehicle 1, therebycontrolling an actuator 146 to maintain a safe distance from thepreceding vehicle. The actuator 146 is an actuator for vehicle controlused to control a brake, a throttle valve, the steering angle and thelike. When collision with a target object is expected, the vehiclecontrol device 140 displays a warning (notification) of the expectedcollision on a display 148 provided in front of a driver, and controlsthe actuator 146 to automatically decelerate the vehicle 1. The vehiclecontrol device 140 can also be integrally implemented with the exteriorenvironment recognition device 130.

FIG. 3 is an explanatory diagram for explaining an effect of environmentlight in the environment recognition system 100. As described above, inthe environment recognition system 100, the image capturing device 110generates the luminance image 124, and the exterior environmentrecognition device 130 uses the luminance based on the luminance image124 to identify which specific object the target object in the detectionregion 122 corresponds to. Light of the light emission body indicated byarrow (A) in FIG. 3 and reflection light of an object as indicated byarrow (B) in FIG. 3 appear in the luminance image 124.

When the image capturing device 110 is provided in the vehicle 1, atransparent body 2 such as windshield exists in the image-capturingdirection of the image capturing device 110, and the luminance image 124is generated via the transparent body 2 by the image capturing device110. Therefore, when the transparent body 2 is dirty or fogged, and thetransparency is reduced. At this occasion, when strong environment lightsuch as sunlight as indicated by arrow (C) in FIG. 3 is emitted from theimage-capturing direction, the entire luminance image 124 thus capturedis affected by the environment light, and, for instance, the environmentlight of arrow (C) is added to the light of the light emission body ofarrow (A), which causes the target object in the luminance image 124 tohave a color different from the original color.

It should be noted that this kind of phenomenon is not limited to thecase where dirt or fog exits on the windshield, and the same can also besaid when dirt or fog exists in the optical system of the lenses.

In this case, the original luminance of the target object can bereproduced by subtracting the effect due to the environment light ofarrow (C), as the amount of correction, from the obtained luminanceimage 124. The amount of correction is obtained, for instance, asfollows: Assuming that a specific object such as a road sign and atraffic light of which RGB ratio is identified in advance is detected,the amount of correction is obtained by subtracting the originalluminance associated with the specific object from the luminance in thetarget portion corresponding to the specific object in the luminanceimage 124. Hereinafter, the specific configuration of the exteriorenvironment recognition device 130 will be explained.

(Exterior Environment Recognition Device 130)

FIG. 4 is a functional block diagram schematically illustratingfunctions of the exterior environment recognition device 130. Asillustrated in FIG. 4, the exterior environment recognition device 130includes an I/F unit 150, a data retaining unit 152, and a centralcontrol unit 154.

The I/F unit 150 is an interface for interactive information exchangewith the image processing device 120 and the vehicle control device 140.The data retaining unit 152 is constituted by a RAM, a flash memory, anHDD and the like, and retains a specific object correspondence table, adirt determination table, a correction reference table, and variouskinds of information required for processing performed by eachfunctional unit explained below. In addition, the data retaining unit152 temporarily retains the luminance image 124 and the distance image126 received from the image processing device 120. The specific objectcorrespondence table, the dirt determination table, and the correctionreference table are used as follows:

FIG. 5 is an explanatory diagram for explaining a specific objectcorrespondence table 200, a dirt determination table 202, and acorrection reference table 204. In this example, firstly, (1) thespecific object correspondence table 200 is used to identify thespecific object on the basis of the luminance image 124. Subsequently,(2) the dirt determination table 202 is used to determine whether or notthe identified specific object is detected with dirt or fog. Then, (3)when it is determined that the identified specific object is detectedwith dirt or fog, the correction reference table 204 is used to derivethe amount of correction which is a difference from the originalluminance of the specific object. The amount of correction thus derivedis applied to detection processing of another specific object in thesame frame and detection processing of all the specific objects in thenext frame and the frames subsequent thereto.

In the specific object correspondence table 200 as illustrated in FIG.5A, multiple specific objects are associated with a luminance range 206indicating a range of luminance representing color (color balance), aheight range 208 indicating a range of height from the road surface, anda width range 210 indicating a range of size of the specific object. Thespecific objects include various objects required to be observed whilethe vehicle runs on the road, such as “traffic light (red),” “trafficlight (blue),” “road sign (blue),” and “road sign (green).” It is to beunderstood that the specific object is not limited to the objects inFIG. 5A. Among the specific objects, for example, the specific object“traffic light (red)” adopts the luminance (R) as the reference value,and is associated with the luminance (G) which is 0.5 times or less ofthe reference value (R), the luminance (B) which is 0.38 times or lessof the reference value (R), the height range 208 which is 4.5 to 7.0 m,and the width range 210 which is 0.05 to 0.2 m. The specific object“road sign (blue)” adopts the luminance (B) as the reference value, andis associated with the luminance (R) which is 0.7 times or less of thereference value (B), the luminance (G) which is 0.8 times or less of thereference value (B), the height range 208 which is 1.5 to 10.0 m, andthe width range 210 which is 0.3 to 1.0 m.

Although not illustrated, each specific object is also associated with acondition unique to the specific object, for instance, information suchas horizontal position, height, and the like with respect to the road.For instance, the specific objects “traffic light (red)” and “road sign(blue)” are associated with, for instance, information indicating thatit is located within the road width in the horizontal direction of thedistance image 126, the relative distance from the vehicle 1 is 40 to 70m, the distance between target portions (variation) is within ±1 m, thedistance between those other than the grouped target portions is 20 m orfarther, and the number of target portions within the group is apredetermined number or equal to or more than a predetermined ratio.When the specific object “traffic light (red)” is constituted by LEDs,operation such as blinking explained later is associated, and the “roadsign (blue)” is associated with, for example, information indicatingthat the size of area of the portion that can be determined to be blueis 50% or more of the entire size of area. In order to identify thespecific object, various kinds of existing techniques can be used. Forexample, a technique disclosed in Japanese Unexamined Patent ApplicationPublication (JP-A) No. 2009-72165 for identifying a specific object onthe basis of the position of a light source in the real world can beused.

In this example, based on the specific object correspondence table 200,any target portion among the target portions in the luminance image 124that satisfies the condition of the luminance range 206 with regard toany given specific object is adopted as a candidate for the specificobject. For instance, when the luminances of the target portion areincluded in the luminance range 206 of the specific object “trafficlight (red)” of the specific object correspondence table 200, the targetportion may be adopted as the candidate for the specific object “trafficlight (red).” Then, the target portions corresponding to the candidate“traffic light (red)” are grouped together to configure the targetobject. When the size of a grouped target object is included in theheight range 4.5 to 7.0 m and the width range 0.05 to 0.2 m of the“traffic light (red),” and the condition unique to the “traffic light(red)” is satisfied, then it is determined to be the specific object“traffic light(red). The target portion determined to be the specificobject is labeled with an identification number unique to the specificobject. Pixels or a block made by collecting pixels may be used as thetarget portion. In this example, pixels are used as the target portionsfor the sake of convenience of explanation.

The dirt determination table 202 as illustrated in FIG. 5B isconstituted by the same items as the specific object correspondencetable 200, but is different in the luminance range 206. The dirtdetermination table 202 is designed to make a determination such that,on the basis of the luminance range 206 which is to be satisfied by thespecific object unless the transparent body has dirt or fog, the rangeother than the luminance range 206 is determined to have dirt or fog.Therefore, for instance, the specific object “traffic light (red)”adopts the luminance (R) as the reference value, and is associated withthe luminance (G) which is 0.35 times or more of the reference value (R)and the luminance (B) which is 0.2 times or more of the reference value(R). The dirt determination table is for the specific object detectedusing the specific object correspondence table 200 of FIG. 5A, andtherefore, in combination with the condition of the specific objectcorrespondence table 200, existence of dirt is determined when theluminance (G) is 0.35 times or more of the reference value (R) and is0.5 times or less thereof or the luminance (B) is 0.2 times or more ofthe reference value (R) and is 0.38 times or less thereof, as a result.

The correction reference table 204 as illustrated in FIG. 5C is alsodifferent in the luminance range 206. The correction reference table 204indicates the original luminance unique to the specific object.Therefore, it is used to derive a difference between the luminance ofthe specific object of which dirt is detected by using the dirtdetermination table 202 and the original luminance of the specificobject, and the derived difference is used as amount of correction forcorrecting the luminance of the specific object.

Back to FIG. 4, the central control unit 154 is comprised of asemiconductor integrated circuit including, for instance, a centralprocessing unit (CPU), a ROM storing programs, and a RAM serving as awork area, and controls an I/F unit 150 and the data retaining unit 152through a system bus 156. In this example, the central control unit 154also functions as a luminance obtaining unit 160, a position informationobtaining unit 162, a specific object provisional determining unit 164,a grouping unit 166, a specific object determining unit 168, atransparency reduction determination unit 172, a correction amountderiving unit 174, and a luminance correction unit 176. In thisembodiment, the position information obtaining unit 162, the specificobject provisional determining unit 164, the grouping unit 166, and thespecific object determining unit 168 function as the specific objectdetection unit for detecting the specific object from the luminanceimage 124 on the basis of the specific object correspondence table 200.

The luminance obtaining unit 160 obtains luminances by target portion(pixels) (luminances of three color phases (red (R), green (G), and blue(B)) per pixel) from the received luminance image 124. When thelater-described luminance correction unit 176 corrects the luminance,the corrected correction luminance is obtained.

The position information obtaining unit 162 uses the stereo method toconvert parallax information for each block in the detection region 122of the distance image 126 received into three-dimensional positioninformation including the width direction x, the height direction y, andthe depth direction z. The parallax information represents a parallax ofeach target portion in the distance image 126, whereas thethree-dimensional position information represents information about therelative distance of each target portion in the real world. Accordingly,a term such as the relative distance and the height refers to a distancein the real world, whereas a term such as a detected distance refers toa distance in the distance image 126. When the parallax information isnot derived per pixel but is derived per block, a calculation may beexecuted per pixel with the parallax information being deemed asparallax information about all the pixels which belong to a block.

FIG. 6 is an explanatory diagram for explaining conversion intothree-dimensional position information by the position informationobtaining unit 162. First, the position information obtaining unit 162treats the distance image 126 as a coordinate system in a pixel unit asillustrated in FIG. 6. In FIG. 6, the lower left corner is adopted as anorigin (0, 0). The horizontal direction is adopted as an i coordinateaxis, and the vertical direction is adopted as a j coordinate axis.Therefore, a pixel having a parallax dp can be represented as (i, j, dp)using a pixel positions i, j and the parallax dp.

The three-dimensional coordinate system in the real world according tothis example will be considered using a relative coordinate system inwhich the vehicle 1 is located in the center. The right side of thedirection in which the vehicle 1 moves is denoted as a positivedirection of X axis, the upper side of the vehicle 1 is denoted as apositive direction of Y axis, the direction in which the vehicle 1 moves(front side) is denoted as a positive direction of Z axis, and thecrossing point between the road surface and a vertical line passingthrough the center of two image capturing devices 110 is denoted as anorigin (0, 0, 0). When the road is assumed to be a flat plane, the roadsurface matches the X-Z plane (y=0). The position information obtainingunit 162 uses numerical expressions (1) to (3) shown below to transformthe coordinate of the pixel (i, j, dp) in the distance image 126 into athree-dimensional point (x, y, z) in the real world.x=CD/2+z·PW·(i−IV)  (1)y=CH+z·PW·(j−JV)  (2)z=KS/dp  (3)

Here, CD denotes an interval (baseline length) between the imagecapturing devices 110, PW denotes a corresponding distance in the realworld to a distance between adjacent pixels in the image, so-called likean angle of view per pixel CH denotes an disposed height of the imagecapturing device 110 from the road surface, IV and JV denote coordinates(pixels) in the image at an infinity point in front of the vehicle 1,and KS denotes a distance coefficient (KS=CD/PW).

Accordingly, the position information obtaining unit 162 derives theheight from the road surface on the basis of the relative distance ofthe target portion and the detection distance in the distance image 126between a point on the road surface located at the same relativedistance as the target portion and the target portion.

The specific object provisional determining unit 164 provisionallydetermines a specific object on the basis of the luminance (or thecorrection luminance) obtained by luminance obtaining unit 160 byreferring to the specific object correspondence table 200 retained inthe data retaining unit 152.

More specifically, the specific object provisional determining unit 164sequentially selects any given specific object from the specific objectsregistered in the specific object correspondence table 200, anddetermines whether the obtained luminances are included in the luminancerange 206 of the specific object sequentially selected. Then, when theluminances are determined to be in the target luminance range 206, anidentification number representing the specific object is assigned tothe target portion, so that a specific object map is generated.

The specific object provisional determining unit 164 sequentiallyexecutes a series of comparisons between the luminances of the targetportions and the luminance ranges 206 of the specific objects registeredin the specific object correspondence table 200. The order of selectingthe specific objects in the specific object table 200 as explained abovealso shows the order of priority. That is, in the specific objectcorrespondence table 200 of FIG. 5A, the comparison processing isexecuted in the following order: “traffic light (red),” “traffic light(blue),” “road sign (blue),” “road sign (green).”

When the comparison is performed according to the above order ofpriority, and as a result, the luminances of the target portion aredetermined to be included in the luminance range 206 of a specificobject of a high order of priority, the comparison processing is nolonger performed for specific objects of a lower order of priority.Therefore, only one identification number representing one specificobject is assigned. This is because multiple specific objects do notoverlap in the real world, and thus a target object that is oncedetermined to be any given specific object is no longer determined to beanother specific object. By exclusively treating the target portions inthis manner, it is possible to avoid redundant specifying processing forthe same target portion that is already provisionally determined to be aspecific object, and the processing load can be reduced.

FIG. 7 is an explanatory diagram for explaining a specific object map220. The specific object map 220 is made by overlaying theidentification numbers of the specific objects on the luminance image124, and the identification number of the specific object is assigned toa position corresponding to the target portion provisionally determinedto be the specific object.

For instance, in a segment 220 a of the specific object map 220, theluminances of target portions 222 corresponding to the light-emittingportions at the right side of the traffic light are included in theluminance range 206 of the specific object “traffic light (red),” andtherefore, an identification number “1” of the specific object “trafficlight (red)” is assigned. FIG. 7 illustrates a figure in whichidentification number “1” is assigned to multiple target portions 222 ofthe luminance image 124. This is, however, a conceptual representationfor the sake of easy understanding. In reality, identification number“1” is registered as data at the target portions 222.

The grouping unit 166 adopts any given target portion provisionallydetermined as a base point, and groups the relevant target portionsprovisionally determined to correspond to a same specific object(attached with a same identification number) of which positiondifferences in the width direction x and in the height direction y arewithin a predetermined range, thereby making the grouped target portionsinto a target object. The predetermined range is represented as adistance in the real world, and can be set at any given value (forexample, 1.0 m). The grouping unit 166 also groups target portions whichare newly added through the grouping processing as a base point andgroups the relevant target portions which are provisionally determinedto correspond to a same specific object and of which positiondifferences in the width direction x and in the height direction y arewithin a predetermined range. Consequently, as long as the distancebetween the target portions provisionally determined to be the samespecific object is within the predetermined range, all of such targetportions are grouped.

In this case, the grouping unit 166 makes the determination using thedistance in the with direction x and the distance in the heightdirection y in the real world, but it may also be possible to use theluminance image 124 and the distance image 126 and make a determinationusing the detection distance (for example, the number of pixels) on theluminance image 124 and the distance image 126. In this case, withoutderiving the distance in the with direction x and the distance in theheight direction y in the real world, for instance, the determination asto whether or not it is within a predetermined range is made on thebasis of only the number of pixels. Note that, in this case, thethreshold value of the predetermined range for grouping is changedaccording to the relative distance of the target portion. As illustratedin FIG. 2 and the like, distant objects and close objects arerepresented in the flat plane in the luminance image 124 and thedistance image 126, and therefore, an object located at a distantposition is represented in a small (short) size and an object located ata close position is represented in a large (long) size. Therefore, forexample, the threshold value of the predetermined range in the luminanceimage 124 and the distance image 126 is set at a small value for adistant target portion, and set at a large value for a close targetportion. Therefore, even when the detection distances are differentbetween a distant position and a close position, the threshold value canbeset appropriately, and the grouping processing can be stablyperformed.

In addition to the difference in the width direction x and thedifference in the height direction y explained above, the grouping unit166 may group target portions of which relative-distance difference inthe depth direction z is within a predetermined range and which areprovisionally determined to correspond to a same specific object. In thereal world, even when target portions are close to each other in thewidth direction x and in the height direction y, the positions (relativedistances) in the depth direction z thereof may be greatly different. Insuch case, the target portions belong to different target objects.Therefore, when any one of the difference of positions in the widthdirection x, the difference of positions in the height direction y, andthe difference of positions (relative distances) in the depth directionz is greatly different, the group of the target portion may be deemed asan independent target object. In so doing, it is possible to performhighly accurate grouping processing.

In the above description, each of the difference in the width directionx, the difference in the height direction y and the difference in thedepth direction z is independently determined, and only when all of themare included within the predetermined range, the target portions aregrouped into the same group. However, grouping processing may beperformed using another calculation. For example, when a square meanvalue √ of the difference in the width direction x, the difference inthe height direction y, and the difference in the depth direction z((difference in the width distance x)²+(difference in the heightdirection y)²+(difference in the depth direction z)²) is included withina predetermined range, target portions may be grouped into the samegroup. With such calculation, distances between target portions in thereal world can be derived accurately, and therefore, grouping accuracycan be enhanced.

When a target object made as a result of grouping processing by thegrouping unit 166 satisfy a predetermined condition such as the heightrange 208 and the width range 210, the specific object determining unit168 determines that the target object is a specific object. For example,as illustrated in FIG. 5A, when the height range 208 is associated withthe specific object correspondence table 200, and the height y withrespect to the road of the target object is included in the height range208 of a specific object provisionally determined with regard to thetarget object on the basis of the specific object correspondence table200, the specific object determining unit 168 determines that the targetobject as the specific object. When the width range 210 is additionallyassociated, then, the specific object determining unit 168 can make adetermination as follows: when the size of the target object (any one ofthe distance in the width direction x and the distance in the widthdirection y of the target object) is included in the width range 210 ofthe specific object provisionally determined with regard to the targetobject on the basis of the specific object correspondence table 200, thespecific object determining unit 168 determines that the target objectis the specific object. It may also be possible to set the width range210 for each of the distance in the width direction x and the distancein the width direction y of the target object. In this case, it isconfirmed that the target object is of a position and a size adequate tobe deemed as a specific object.

When information unique to the specific object such as horizontalposition, height, and the like with respect to the road is alsoassociated with each specific object, the specific object determiningunit 168 may determine the specific object only when the condition issatisfied. For instance, when a candidate of the specific object is the“road sign (blue),” then it is determined whether the followingconditions are satisfied: the horizontal position with respect to theroad is located within the road width in the horizontal direction of thedistance image 126 and the horizontal distance x from the vehicle 1 is40 m to 70 m; the distance other than the grouped target portion is 20 mor longer; the number of target portions in the group is equal to orlarger than a predetermined number or a predetermined ratio; and thesize of area of the portion that can be determined to be blue is 50% ormore of the entire size of area.

When the specific object has other features, the specific object may bedetermined using the features as the condition. For instance, the“traffic light (red)” is constituted by light emitting diodes (LEDs) orlamps, and when it is constituted by LEDs, the LEDs are blinking at acycle that cannot be perceived by the human eyes (for example, 100 Hz).Therefore, the specific object determining unit 168 can determine thespecific object “traffic light (red)” on the basis of temporal change ofthe luminance of the target portion in the luminance image 124 obtainedwithout synchronization with the blinking timing of the LEDs.

The transparency reduction determination unit 172 compares the luminanceof the target portion corresponding to the specific object determined(detected) and the luminance range 206 associated with the specificobject with the dirt determination table 202, and determines a reductionin transparency (dirt or fog) of the transparent body 2 located in theimage-capturing direction (the image-capturing direction of the imagecapturing device 110) of the luminance image 124.

FIG. 8 is an explanatory diagram for explaining a determination targetof the transparency reduction determination unit 172. The transparencyreduction determination unit 172 determines the reduction intransparency of the transparent body 2 for each divided region obtainedby dividing the luminance image 124 (detection region 122) into multipleregions. In this example, for instance, as illustrated in FIG. 8,divided regions 230 are set by equally dividing the luminance image 124into five regions in the horizontal direction and equally dividing theluminance image 124 into three regions in the vertical direction. Inthis example, the luminance image 124 is divided into 5-by-3 regions,but the number of regions divided may be set to any number, includingzero. In this example, the luminance image 124 is equally divided in thehorizontal direction and the vertical direction, but it may be dividedinto various sizes. For instance, the size of area of the centraldivided region 230 of the luminance image 124 may to be smaller thanthat at the end portion side.

When, for each divided region 230, the luminance of the target portioncorresponding to the determined specific object is included in theluminance range 206 associated with the specific object in the dirtdetermination table 202, then the transparency reduction determinationunit 172 determines that the transparency of the transparent body 2corresponding to the divided region 230 including the target portion isreduced. For instance, even if the specific object provisionaldetermining unit 164 determines that the specific object “road sign(blue)” is included in the luminance range 206 on the basis of thespecific object correspondence table 200, the transparency of thetransparent body 2 is determined to be reduced when the luminance isincluded in the luminance range 206 of the dirt determination table 202.

As described above, by dividing the luminance image 124 into multipledivided regions 230, reduction in the transparency of the transparentbody 2 can be determined independently with a fine range, and this canprevent correction of the luminance even in a divided region 230 ofwhich transparency has not been reduced.

Instead of the transparency reduction determination unit 172, it is alsobe possible to use a transparency reduction detection unit that detects,with hardware, a reduction in the transparency of the transparent body 2located in the image-capturing direction of the luminance image 124.Various kinds of existing technique can be used for such detection unit.

When the transparency reduction determination unit 172 and thetransparency reduction detection unit detect a reduction in thetransparency of the transparent body 2, the correction amount derivingunit 174 looks up the correction reference table 204, derives the amountof correction on the basis of difference between the luminance of thetarget portion of the luminance image 124 associated with the specificobject determined (detected) and the luminance associated with thespecific object in the correction reference table 204.

Like the transparency reduction determination unit 172, the correctionamount deriving unit 174 also derives the amount of correction perdivided region 230 obtained by dividing the luminance image 124(detection region 122) into multiple regions. By deriving the amount ofcorrection for each of the multiple divided regions 230, the luminancecan be appropriately corrected for each of the divided regions 230.

FIG. 9 is an explanatory diagram for explaining processing of thecorrection amount deriving unit 174. For example, suppose a case wherethe luminances of any given target portion obtained by the luminanceobtaining unit 160 are R=90, G=100, and B=150, and a close target objectto the target portion is determined to be the specific object “road sign(blue)” on the basis of the specific object correspondence table 200.

In this case, as illustrated in FIG. 9A, the range of the specificobject “road sign (blue) in the specific object correspondence table200, that is, the reference value, is the luminance (B), and theluminance (R) is 0.6 times of the reference value (B), and the luminance(G) is 0.67 times of the reference value (B), the conditions “R≦B×0.7,”“G≦B×0.8” are satisfied. However, as illustrated in FIG. 9B, theluminances of the target portion are such that the range of the “roadsign (blue)” in the dirt determination table 202, that is, the referencevalue is the luminance (B), and the luminance (R) is 0.55 times or moreof the reference value (B), and the luminance (G) is 0.65 times or moreof the reference value (B). Therefore, the transparency reductiondetermination unit 172 determines that, in the divided region 230including the target object, the transparency of the transparent body 2is reduced.

Following the determination of the transparency reduction determinationunit 172, the correction amount deriving unit 174 derives the amount ofcorrection. Firstly, the correction amount deriving unit 174 reduces theluminances of each color of the target portion by −1 in a range wherethe calculation result is 0 to 255, and a difference from a value whenthe result is the same as the luminance of the “road sign (blue)” of thecorrection reference table 204 is adopted as the amount of correction.

For instance, as illustrated in FIG. 9C, the luminances R=89, G=99,B=149 obtained by reducing the luminances of the target portion R=90,G=100, B=150 have not yet reached the luminancse of the “road sign(blue)” of the correction reference table 204. However, as illustratedin FIG. 9D, when the luminances R=40, G=50, B=100 are reduced 50 times,R/B becomes 0.4, and G/B becomes 0.5, and this matches the luminances ofthe “road sign (blue)” in the correction reference table 204. In thiscase, the amount of correction is 50 which is the difference.

When there are multiple luminances of the target as described above, thedifference when any one of them satisfies the condition may be adoptedas the amount of correction, or the difference when both of them areless than the condition (for example, in the case of the “road sign(blue),” R/B<0.4, G/B<0.5, and in the case of the “traffic light (red),”G/R<0.25, B/R<0.1″) may be adopted as the amount of correction.

When the luminances are reduced until both of them are less than thecondition, any one of the luminances becomes a negative value, thedifference of the luminances when the absolute value of the average(central value) of differences between the luminances (ratio) when theluminances of the target portion are reduced and the luminances (ratio)of the correction reference table 204 are the minimum may be adopted asthe amount of correction. Hereinafter, details will be described usingthe specific object “traffic light (red)” as an example.

FIG. 10 is another explanatory diagram for explaining processing of thecorrection amount deriving unit 174. For instance, suppose that theluminances of any given target portion obtained by the luminanceobtaining unit 160 are R=120, G=45, B=18, and a target object inproximity to the target portion is determined to be the specific object“road sign (red)” on the basis of the specific object correspondencetable 200.

In this case, as illustrated in FIG. 10A, the range of the specificobject “road sign (red)” in the specific object correspondence table200, that is, the reference value, is the luminance (R), and theluminance (G) is 0.375 times of the reference value (R), and theluminance (B) is 0.15 times of the reference value (R), the conditions“G≦R×0.5,” “B≦R×0.38” are satisfied. However, as illustrated in FIG.10B, the luminances of the target portion are such that the range of the“traffic light (red)” in the dirt determination table 202, that is, thereference value is the luminance (R), and the luminance (G) is 0.35times or more of the reference value (R). Therefore, the transparencyreduction determination unit 172 determines that, in the divided region230 including the target object, the transparency of the transparentbody 2 is reduced.

Upon the determination of the transparency reduction determination unit172, the correction amount deriving unit 174 derives the amount ofcorrection. Firstly, the correction amount deriving unit 174 reduces theluminances of each color of the target portion by −1 in a range wherethe calculation result is 0 to 255, and tries so that the result becomesthe same as the luminances of the “traffic light (red)” of thecorrection reference table 204, that is, “G=R×0.25,” “B=R×0.1.”

However, when the luminances are reduced by one, both of the conditionsshould be satisfied at the same time, but when one of them satisfies thecondition, the other of them may not necessarily satisfy the condition.Accordingly, the correction amount deriving unit 174 may adopt, as theamount of correction, the difference of the luminances when the absolutevalue of the average of the difference between the luminances obtainedby reducing the luminances of the target portion and the luminances ofthe correction reference table 204 is the minimum.

For example, the absolute value of the average of the difference fromthe luminances of the correction reference table 204 gradually decreasesdue to decrement, and when the luminances of the target portion obtainedthrough decrement are R=108, G=33, B=6 as illustrated in FIG. 10C, thenthe absolute value of the average of the difference from the luminancesof the correction reference table 204 becomes 0.006. Likewise, asillustrated in FIG. 10D, when the luminancse of the target portionobtained through decrement re R=107, G=32, B=5, the absolute value ofthe average of the difference from the luminances of the correctionreference table 204 becomes 0.002, and when the luminances of the targetportion obtained through decrement are R=106, G=31, B=4 as illustratedin FIG. 10E, the absolute value of the average of the difference fromthe luminances of the correction reference table 204 becomes 0.010. Theabsolute value of the average of the difference from the luminances ofthe correction reference table 204 gradually increases due to decrement.That is, the absolute value 0.002 of the average of the difference fromthe luminances of the correction reference table 204 where R=107, G=32,B=5 are the minimum values, and the amount of correction at this momentis 13 which is the difference from the luminance before the decrement.

As described above, the amount of correction is derived, but asdescribed with reference to FIG. 3, the change in the luminances iscaused by the effect of the environment light of arrow (C). The effectof the environment light is represented by a product of the transparencyand the intensity of the incident light (such as sunlight). Thetransparency is in accordance with dirt or fog on the transparent body2, and the value thereof is substantially a fixed value, but theintensity of the incident light is changed in a proportional manner inaccordance with the exposure time of the image capturing device 110.

Therefore, when the correction amount deriving unit 174 derives theamount of correction, the exposure time when the amount of correction isderived is obtained, and derives the basic amount of correction(corresponding to the transparency) obtained by dividing the amount ofcorrection by the exposure time. The value obtained by multiplying thebasic amount of correction by the exposure time of luminance image 124of the correction target is the amount of correction of the luminanceimage 124 of the correction target.

As described above, the correction amount deriving unit 174 derives theamount of correction per divided region 230. The amount of correctionderives when there is a specific object in a divided region 230.Therefore, when there is no specific object in a divided region 230, thecorrection is not made even if the transparency of the transparent body2 is reduced. Therefore, when there are not only a divided region 230for which the amount of correction is derived but also a divided region230 for which the amount of correction is not derived in a mixed manner,the correction amount deriving unit 174 derives the amount of correctionof the divided region 230 for which the amount of correction is notderived, on the basis of the amount of correction of the divided region230 for which the amount of correction is derived. For instance, thecorrection amount deriving unit 174 averages the amounts of correctionsof one or more divided regions 230 for which the amount of correction isderived, and adopts the average value as the amount of correction of thedivided region 230 for which the amount of correction is not derived.However, the derived average value is not reflected in the dividedregion 230 for which the amount of correction is derived.

When the entire transparent body 2 is dirty or fogged, only a portion ofthe divided region 230 is not corrected, and the above configuration canprevent the identifying accuracy of the specific object from reducing.

The amount of correction changes over time and changes greatly.Accordingly, the correction amount deriving unit 174 derives the timeaverage value of the amount of correction derives this time and theamount of correction previously derived in the same detection region 122or the same divided region 230, and adopts the time average value as theamount of correction again. The time average value may be a valueobtained by simply averaging the amount of correction derived this timeand the previously derived amount of correction for a predeterminednumber of times, or a value derived by one or more degrees of low passfilters (LPFs) having predetermined time constants.

With such configuration, the change of the amount of correction overtime can be alleviated, and the change of the amount of correction canbe suppressed.

The luminance correction unit 176 corrects the luminance image 124 onthe basis of the amount of correction derived. As described above, thecorrected luminance image is used to identify a specific object otherthan the specific object used for calculating the amount of correction(for instance, when a calculation is performed on the basis of theluminance information about the road sign (blue), the correctedluminance image is used to identify a specific object other than theroad sign (blue), such as tail lamps and a traffic light). In the nextframe and frames subsequent thereto, the luminance correction unit 176corrects the received luminance image 124 on the basis of the amount ofcorrection, and causes the luminance obtaining unit 160 to obtain thecorrected correction luminance. Such luminance correction may beexecuted either constantly or only when the transparency reductiondetermination unit 172 and the transparency reduction detection unitdetects reduction in the transparency of the transparent body 2.

Like the transparency reduction determination unit 172, the luminancecorrection unit 176 also corrects the luminance per divided region 230obtained by dividing the luminance image 124 (the detection region 122)into multiple regions. As described above, the luminance can beappropriately corrected for each of the multiple divided regions 230. Inthis example, the amount of correction for the divided region 230 forwhich the amount of correction is not derived is also derived on thebasis of the divided region 230 for which the amount of correction isderived, and the luminance of the divided region 230 is corrected, butit is possible not to correct at all the divided region 230 for whichthe amount of correction is not derived.

As described above, the correction amount deriving unit 174 derives thebasic amount of correction obtained by dividing the amount of correctionby the exposure time when the amount of correction is derived.Therefore, the luminance correction unit 176 performs correction on thebasis of the amount of correction obtained by multiplying the basicamount of correction derived by the correction amount deriving unit 174by the exposure time of the luminance image 124 of the correctiontarget. In this manner, regardless of the change of the exposure time,the luminance can be appropriately corrected.

Therefore, the exterior environment recognition device 130 can extract,from the luminance image 124, one or more target objects as specificobjects, and the information can be used for various kinds of controls.For instance, when the specific object “traffic light (red)” isextracted, this indicates that the target object is a fixed object thatdoes not move, and when the target object is a traffic light for thedriver's vehicle, this indicates that the vehicle 1 has to stop ordecelerate. For another instance, which is not described in the above,the existence of a preceding vehicle running together with the vehicle 1can be recognized by extracting a specific object “tail lamp (red).”

(Environment Recognition Method)

Hereinafter, the specific processing performed by the exteriorenvironment recognition device 130 will be explained with reference tothe flowchart of FIGS. 11 to 16. FIG. 11 illustrates an overall flow ofinterrupt processing when the image processing device 120 transmits thedistance image (parallax information) 126. FIGS. 12 to 16 illustratesubroutines therein. In this description, pixels are used as targetportions, and the lower left corners of the luminance image 124 and thedistance image 126 are origins. The processing is performed according tothe environment recognition method in a range of 1 to 600 pixels in thehorizontal direction of the image and 1 to 180 pixels in the verticaldirection of the image. In this description, divided regions 230 withfive regions in the horizontal direction by three regions in thevertical direction are prepared, and each divided region 230 hashorizontal 120 pixels by 60 pixels. The number of target specificobjects to be checked is considered to be four.

As illustrated in FIG. 11, when an interrupt occurs according to theenvironment recognition method in response to reception of the distanceimage 126, the luminance image 124 obtained from the image processingdevice 120 is referred to, and a specific object map 220 is generated onthe basis of the target object correspondence table 200 (S300). With thespecific object map 220, the specific objects provisionally determinedare made into a group (S302), and the grouped target objects aredetermined as a specific object (S304).

Subsequently, reduction in the transparency of the transparent body 2 inthe image-capturing direction of the luminance image 124 is examined(S308), and when the amount of correction of the target portion of theluminance image 124 is derived (S310), the luminances of the targetportion of the luminance image 124 are corrected on the basis of theamount of correction derived (S312). At this occasion, the luminancecorrection unit 176 performs correction on the basis of the amount ofcorrection obtained by multiplying the basic amount of correctionderived by the correction amount deriving unit 174 by the exposure timeof the luminance image 124 of the correction target. Hereinafter, theprocessing will be explained in a more specific manner.

(Specific Object Map Generating Processing S300)

As illustrated in FIG. 12, the specific object provisional determiningunit 164 initializes (substitutes “0” to) a vertical variable j forspecifying a target portion (pixel) (S400). Subsequently, the specificobject provisional determining unit 164 adds “1” to (increments by 1)the vertical variable j, and initializes (substitutes “0” to) ahorizontal variable i (S402). Then, the specific object provisionaldetermining unit 164 adds “1” to the horizontal variable i, andinitializes (substitutes “0” to) a specific object variable m (S404).Here, the horizontal variable i and the vertical variable j are providedto execute the specific object map generating processing on all of the600 by 180 pixels, and the specific object variable m is provided tosequentially compare four specific objects for each pixel.

The specific object provisional determining unit 164 causes theluminance obtaining unit 160 to obtain the luminance of the pixel (i, j)as a target portion from the luminance image 124 (S406), adds “1” to thespecific object variable m (S408), obtains the luminance range 206 ofthe specific object (m) (S410), and determines whether or not theluminances of the pixel (i, j) are included in the luminance range 206of the specific object (m) (S412).

When the luminances of the pixel (i, j) are included in the luminancerange 206 of the specific object (m) (YES in S412), the specific objectprovisional determining unit 164 assigns an identification number prepresenting the specific object (m) to the pixel so as to be expressedas the pixel (i, j, p) (S414). In this manner, the specific object map220 is generated, in which an identification number p is given to eachpixel in the luminance image 124. When the luminances of the pixel (i,j) are not included in the luminance range 206 of the specific object(m) (NO in S412), a determination is made as to whether or not thespecific object variable m is equal to more than 4, which is the maximumnumber of specific objects (S416). When the specific object variable mis less than the maximum value (NO in S416), the processing is repeatedfrom the increment processing of the specific object variable m in stepS408. When the specific object variable m is equal to or more than themaximum value (YES in S416), which means that there is no specificobject corresponding to the pixel (i, j), and the processing in stepS418 subsequent thereto is performed.

Then, the specific object provisional determining unit 164 determineswhether or not the horizontal variable i is equal to or more than 600,which is the maximum value of the pixel number in the horizontaldirection (S418), and when the horizontal variable i is less than themaximum value (NO in S418), the processing is repeated from theincrement processing of the horizontal variable i in step S404. When thehorizontal variable i is equal to or more than the maximum value (YES inS418), the specific object provisional determining unit 164 determineswhether or not the vertical variable j is equal to or more than 180,which is the maximum value of the pixel in the vertical direction(S420). Then, when the vertical variable j is less than the maximumvalue (NO in S420), the processing is repeated from the incrementprocessing of the vertical variable j in step S402. When the verticalvariable j is equal to or more than the maximum value (YES in S420), thespecific object map generating processing S300 is terminated. Therefore,the specific object corresponding to each pixel is provisionallydetermined.

(Grouping Processing S302)

As illustrated in FIG. 13, the grouping unit 166 refers to thepredetermined range to group the target portions (S450), and initializes(substitutes “0” to) the vertical variable j for specifying a targetportion (pixel) (S452). Subsequently, the grouping unit 166 adds “1” tothe vertical variable j, and initializes (substitutes “0” to) thehorizontal variable i (S454). Then, the grouping unit 166 adds “1” tothe horizontal variable i (S456).

The grouping unit 166 obtains a pixel (i, j, p, dp, x, y, z) as thetarget portion from the luminance image 124 (S458). Then, adetermination is made as to whether an identification number p of thespecific object is assigned to the pixel (i, j, p, dp, x, y, z) (S460).When the identification number p is assigned (YES in S460), the groupingunit 166 determines whether or not there is another pixel (i, j, p, dp,x, y, z) assigned the same identification number p within apredetermined range from the coordinate (x, y, z) in the real world ofthe pixel (i, j, p, dp, x, y, z) (S462).

When there is another pixel (i, j, p, dp, x, y, z) assigned the sameidentification number (YES in S462), the grouping unit 166 determineswhether the group number g is given to any of all the pixels within thepredetermined range including the pixel under determination (S464). Whenthe group number g is given to any of them (YES in S464), the groupingunit 166 assigns a value to all of the pixels included within thepredetermined range and all of the pixels to which the same group numberg is given, the value being a smaller one of the smallest group number gamong the group numbers g given thereto or the smallest value of numbersthat have not yet used as the group numbers g, so as to expressed as apixel (i, j, p, dp, x, y, z, g) (S466). When the group number g is givento none of them (NO in S464), the smallest value of numbers g that havenot yet used as the group numbers g is newly assigned to all the pixelswithin the predetermined range including the pixel under determination(S468).

In this manner, when there are multiple target portions that have a sameidentification number p within the predetermined range, grouping processis performed by assigning one group number g. If a group number g isgiven to none of the multiple target portions, a new group number g isassigned, and if a group number g is already given to any one of them,the same group number g is assigned to the other target portions.However, when there are multiple group numbers g in multiple targetportions, the group numbers g of all the target portions are replacedwith one group number g so as to treat the target portions as one group.

In the above description, the group numbers g of not only all the pixelsincluded within the predetermined range but also all the pixels to whichthe same group number g is given are changed at a time. The primaryreason for this is to avoid dividing the group already unified bychanging the group numbers g. In addition, a smaller one of the smallestgroup number g or the smallest value of numbers that have not yet usedas the group numbers g is employed in order to avoid making a skippednumber as much as possible upon group numbering. In so doing, themaximum value of the group number g does not become unnecessarily large,and the processing load can be reduced.

When the identification number p is not assigned (NO in S460), or whenthere is no other pixel that has the same identification number p (NO inS462), the processing in step S470 subsequent thereto is performed.

Subsequently, the grouping unit 166 determines whether or not thehorizontal variable i is equal to or more than 600, which is the maximumvalue of pixel number in the horizontal direction (S470). When thehorizontal variable i is less than the maximum value (NO in S470), theprocessing is repeated from the increment processing of the horizontalvariable i in step S456. When the horizontal variable i is equal to ormore than the maximum value (YES in S470), the grouping unit 166determines whether or not the vertical variable j is equal to or morethan 180, which is the maximum value of pixel number in the verticaldirection (S472). When the vertical variable j is less than the maximumvalue (NO in S472), the processing is repeated from the incrementprocessing of the vertical variable j in step S454. When the verticalvariable j is equal to or more than the maximum value (YES in S472), thegrouping processing S302 is terminated.

(Specific Object Determining Processing S304)

As illustrated in FIG. 14, the specific object determining unit 168initializes (substitutes “0” to) a group variable k for specifying agroup (S500). Subsequently, the specific object determining unit 168adds “1” to the group variable k (S502).

The specific object determining unit 168 determines whether or not thereis a target object of which group number g is the group variable k fromthe luminance image 124 (S504). When there is such target object (YES inS504), the specific object determining unit 168 calculates the heightand the size of the target object to which the group number g is given(S506). Then, a determination is made as to whether or not thecalculated height and the calculated size are included within the heightrange 208 and the width range 210 of a specific object represented bythe identification number p assigned to the target object of which groupnumber g is the group variable k, and whether or not the conditionunique to the specific object is satisfied (S508).

When the height and the size are included within the height range 208and the width range 210 of the specific object represented by theidentification number p, and the condition unique to the specific objectis satisfied (YES in S508), the specific object determining unit 168determines that the target object is the specific object (S510). Whenthe height and the size are not included within the width range 208 andthe width range 210 of the specific object represented by theidentification number p, or the condition unique to the specific objectis not satisfied (NO in S508), or, when there is no target object ofwhich group number g is the group variable k (NO in S504), theprocessing in step S512 subsequent thereto is performed.

Subsequently, the specific object determining unit 168 determineswhether or not the group variable k is equal to or more than the maximumvalue of group number set in the grouping processing S302 (S512). Then,when the group variable k is less than the maximum value (NO in S512),the processing is repeated from the increment processing of the groupvariable k in step S502. When the group variable k is equal to or morethan the maximum value (YES in S512), the specific object determiningprocessing S304 is terminated. As a result, the grouped target objectsare determined to be the specific object.

(Transparency Reduction Determination Processing S308)

As illustrated in FIG. 15, the transparency reduction determination unit172 stores the target object determined to be the specific object in thespecific object determining processing S304 (hereinafter referred to asspecific object) to a predetermined storage region (S550). Then, thetransparency reduction determination unit 172 determines whether thespecific object remains in the storage region (S552). As a result, whenthe specific object remains (YES in S552), one specific object isextracted, and the specific object is deleted from the storage region(S554). When the specific object does not remain (NO in S552), thetransparency reduction determination processing S308 is terminated.

Subsequently, the transparency reduction determination unit 172determines whether the luminances of the target portion corresponding tothe extracted specific object are included in the luminance range 206associated with the specific object in the dirt determination table 202(S556). As a result, if the luminances are included in the luminancerange 206 (YES in S556), the transparency of the transparent body 2 isdetermined to have been reduced with regard to the divided region 230including the specific object (S558), the processing in step S552 andsubsequent steps are repeated. When the luminances are not included inthe range of the luminance (NO in S556), no processing is performed, andthe processing in step S552 and subsequent steps are repeated. In thismanner, the reduction in the transparency of the transparent body 2 withregard to the specific object is determined.

(Correction Amount Deriving Processing S310)

As illustrated in FIG. 16, the correction amount deriving unit 174stores the specific object of which transparency is determined to havebeen reduced in the transparency reduction determination processing S308to a predetermined storage region (S600). Then, the correction amountderiving unit 174 determines whether the specific object remains in thestorage region (S602). As a result, when the specific object remains(YES in S602), one specific object is extracted, and the specific objectis deleted from the storage region (S604). When the specific object doesnot remain (NO in S602), the correction amount deriving processing S310is terminated.

Subsequently, the correction amount deriving unit 174 reduces theluminances of the target portion corresponding to the extracted specificobject (S606), and a determination is made as to whether the resultantluminances have reached the luminance range 206 of the specific objectin the correction reference table 204 (S608). If the resultantluminances are determined to have reached the luminance range 206 (YESin S608), the processing in step S610 subsequent thereto is performed.If the resultant luminances are determined not to have reached theluminance range 206, the processing in step S606 and subsequent stepsare repeated.

Then, the correction amount deriving unit 174 defines, as the amount ofcorrection for the divided region 230 including the specific object, adifference from the value before the decrement when the luminance range206 has been reached (S610). The correction amount deriving unit 174 mayderive the time average value of the amount of correction derived thistime and the amount of correction previously derived in the samedetection region 122 or the same divided region 230, and adopt the timeaverage value as the amount of correction again. When the correctionamount deriving unit 174 derives the amount of correction, the exposuretime when the amount of correction is derived is obtained, and alsoderives the basic amount of correction obtained by dividing the amountof correction by the exposure time.

Subsequently, the correction amount deriving unit 174 determines whetheror not the amount of correction has been set in all the divided regions230 (S612). If there is a divided region 230 for which the amount ofcorrection is not set (NO in S612), the correction amount deriving unit174 averages the amounts of corrections of one or more divided regions230 for which the amount of correction is derived, and adopts theaverage value as the amount of correction of the divided region 230 forwhich the amount of correction is not derived (S614), and the processingin step S602 and subsequent steps are repeated. If the amount ofcorrection has been set in all the divided regions 230 (YES in S612), noprocessing is performed, and the processing in step S602 and subsequentsteps are repeated. In this manner, the amount of correction is set foreach divided region 230.

As described hereinabove, according to the exterior environmentrecognition device 130, existence of dirt or fog on the windshield andthe optical component of the onboard camera can be detected, and theimage can be appropriately recognized using the color information evenunder such environment.

In addition, a program for allowing a computer to function as theexterior environment recognition device 130 is also provided as well asa storage medium such as a computer-readable flexible disk, amagneto-optical disk, a ROM, a CD, a DVD, and a BD storing the program.Here, the program means a data processing function described in anylanguage or description method.

The present invention is not limited to the above-described example. Itwill be apparent to those skilled in the art that various changes may bemade without departing from the scope of the invention.

In the above example, the luminance obtaining unit 160, the positioninformation obtaining unit 162, the specific object provisionaldetermining unit 164, the grouping unit 166, the specific objectdetermining unit 168, the transparency reduction determination unit 172,the correction amount deriving unit 174, and the luminance correctionunit 176 are configured to be operated by the central control unit 154with software. However, the functional units may be configured withhardware.

In the above example, the transparency reduction determination unit 172detects a reduction in the transparency of the transparent body 2, andthereafter the luminance correction unit 176 corrects the luminance ofthe target portion on the bass of the amount of correction by thedetermination result. However, the present invention is not limitedthereto. Only the transparency reduction determination unit 172 maydetermine a reduction in the transparency of the transparent body 2. Forexample, when the transparency reduction determination unit 172 detectsa reduction in the transparency of the transparent body 2, this may beinformed to a driver and a passenger of the vehicle 1. In addition,without making a determination, the luminance correction unit 176 maycorrect the luminance. For instance, the specific object correspondencetable 200 may be used to identify a specific object, and thereafter onlythe correction reference table 204 may be used to make correctionwithout causing the dirt determination table 202 to determine areduction in the transparency of the transparent body 2.

In the above example, for the sake of convenience, the specific objectcorrespondence table 200 is described with the traffic light (red), thetraffic light (blue), the road sign (blue), and the road sign (green)are used for the description, but the present invention is not limitedthereto. Any specific object which exists on the road and of which RGBvalues are substantially constant may be adopted as the target. Forinstance, the RGB values of a turn signal can be roughly identified, andalthough there is some variation, the RGB values of a tail lamp can beroughly identified. Thus, they can be used as specific objects.

The steps of the environment recognition method in the above example donot necessarily need to be processed chronologically according to theorder described in the flowchart. The steps may be processed inparallel, or may include processing using subroutines.

The present invention can be used for an exterior environmentrecognition device and an environment recognizing method for recognizinga target object based on the luminances of the target object in adetection region.

The invention claimed is:
 1. An exterior environment recognition devicefor recognizing the environment outside of a subject vehicle, on thebasis of a color image captured by an onboard camera, the exteriorenvironment recognition device comprising: a specific object detectionunit to detect a specific object on the basis of the color image; a dataretaining unit to associate and retain the specific object and aluminance range indicating the color of the specific object, and toassociate and retain the specific object and the original luminance ofthe specific object; a transparency reduction determination unit tocompare a luminance of the color image of the specific object and aluminance range associated with the specific object, and to determine areduction in transparency of a transparent body located in animage-capturing direction of the onboard camera; a correction amountderiving unit to derive the amount of correction on the basis of adifference between luminance in the color image of the specific objectand the original luminance associated with the specific object; and aluminance correction unit to correct the luminance of the target portionof the color image on the basis of the amount of correction thusderived, wherein the specific object detection unit detects the specificobject on the basis of the corrected color image.
 2. The exteriorenvironment recognition device according to claim 1, wherein thecorrection amount deriving unit derives a basic amount of correctionobtained by dividing the derived amount of correction by an exposuretime of the color image; and the luminance correction unit corrects theluminance of the target portion of the color image on the basis of theamount of correction obtained by multiplying the basic amount ofcorrection by the exposure time of the color image of correction target.3. The exterior environment recognition device according to claim 1,wherein the correction amount deriving unit derives the amount ofcorrection per divided region obtained by dividing the color image intomultiple regions.
 4. The exterior environment recognition deviceaccording to claim 2, wherein the correction amount deriving unitderives the amount of correction per divided region obtained by dividingthe color image into multiple regions.
 5. The exterior environmentrecognition device according to claim 3, wherein the correction amountderiving unit derives an amount of correction of a divided region forwhich the amount of correction is not derived, on the basis of an amountof correction of a divided region for which an amount of correction isderived.
 6. The exterior environment recognition device according toclaim 4, wherein the correction amount deriving unit derives an amountof correction of a divided region for which the amount of correction isnot derived, on the basis of an amount of correction of a divided regionfor which an amount of correction is derived.
 7. The exteriorenvironment recognition device according to claim 1, wherein thecorrection amount deriving unit adopts again, as the amount ofcorrection, a time average value of the amount of correction derived onthe basis of a difference between luminance of the color image of thespecific object and the original luminance associated with the specificobject and the amount of correction previously derived in the samedetection region or the same divided region.
 8. The exterior environmentrecognition device according to claim 2, wherein the correction amountderiving unit adopts again, as the amount of correction, a time averagevalue of the amount of correction derived on the basis of a differencebetween luminance of the color image of the specific object and theoriginal luminance associated with the specific object and the amount ofcorrection previously derived in the same detection region or the samedivided region.
 9. The exterior environment recognition device accordingto claim 1, wherein the specific object detection unit detects thespecific object on the basis of temporal change of the luminance of thecolor image over time.
 10. The exterior environment recognition deviceaccording to claim 2, wherein the specific object detection unit detectsthe specific object on the basis of temporal change of the luminance ofthe color image over time.
 11. The exterior environment recognitiondevice according to claim 3, wherein the specific object detection unitdetects the specific object on the basis of temporal change of theluminance of the color image over time.
 12. The exterior environmentrecognition device according to claim 3, wherein the correction amountderiving unit adopts again, as the amount of correction, a time averagevalue of the amount of correction derived on the basis of a differencebetween luminance of the color image of the specific object and theoriginal luminance associated with the specific object and the amount ofcorrection previously derived in the same detection region or the samedivided region.
 13. The exterior environment recognition deviceaccording to claim 4, wherein the correction amount deriving unit adoptsagain, as the amount of correction, a time average value of the amountof correction derived on the basis of a difference between luminance ofthe color image of the specific object and the original luminanceassociated with the specific object and the amount of correctionpreviously derived in the same detection region or the same dividedregion.
 14. The exterior environment recognition device according toclaim 4, wherein the specific object detection unit detects the specificobject on the basis of temporal change of the luminance of the colorimage over time.